Skip to content

fix(deceptive-bytes): align parsed fields and correlation consumers - #2666

Merged
osmontero merged 7 commits into
utmstack:v11from
kryonsx:codex/v11-deceptive-bytes-review-20260923
Sep 25, 2026
Merged

osmontero merged 7 commits into
utmstack:v11from
kryonsx:codex/v11-deceptive-bytes-review-20260923

Conversation

@kryonsx

@kryonsx kryonsx commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

The Deceptive Bytes filter discarded commands at finalization, emitted errors for absent optional KV inputs, and wrote a nonstandard blocked outcome. Several rules also compared KV values with native booleans, although KV stores strings, and four history rules searched on placeholders an event may lack. This draft preserves commands as origin.command, guards the three optional KV steps, maps explicit blocked/prevented outcomes to denied, and changes seven existing rules. The original vendor action remains available.

Five rules use string-boolean comparisons that also accept native true in go-sdk v1.1.36; the data theft, advanced threat tactic, zero-day and ransomware rules now require the fields their history searches use (origin.ip, and log.tacticName or log.processName; the ransomware rule log.process and log.source_ip). Otherwise a missing placeholder fails the search, and five failures switch the rule off with a Circuit Breaker alert. Existing event labels, thresholds, time windows, attribution and severity-letter mappings are retained.

Re-validated on the latest versions (2026-09-24): official v11 d2479c1a (merged into this branch, no conflicts; plugins/alerts now pins go-sdk v1.1.36) and EventProcessor main 8a3ade7, the same source revision as the newest published engine image (eventprocessor:v11.2.14), compiled natively here. Since go-sdk v1.1.35 the field-name sanitizer keeps underscores, so KV stores the vendor's own keys (log.event_type, log.source_ip). An earlier revision of this draft respelled 18 such names in six rules without underscores, as the older SDK stored them; on the latest engine those names never match, so this revision restores the original names. Lateral movement, living off the land and privilege escalation are again identical to v11; ransomware differs from v11 only by its placeholder guard.

Validation (EventProcessor 8a3ade7, go-sdk v1.1.36):

  • Original/corrected raw parser comparison: four fabricated input pairs verify command retention, present-KV preservation, blocked outcome normalization and missing-source error removal. The results equal those of the earlier build.
  • KV names: the committed fabricated lines carry all 18 underscored keys, and the playground stored every one with its underscore and none without it.
  • Rules in the playground: with the respelled names the Living Off The Land positive raised no alert; with the restored names the Living Off The Land, nation-state and privilege-escalation positives each raised exactly one intended alert, and the near-miss and other lines none.
  • go-sdk v1.1.36 rule replay over those events, plus copies given an origin.ip and synthetic events for the boolean rules: each rule matched only its intended case, every history placeholder resolved on those matches, and the respelled names matched nothing (89/89 checks). With the ransomware guard, two copies of the ransomware line that each lack one placeholder field no longer match (65/65 checks); the unguarded rule matched both with unresolved placeholders.
  • The history-guard test covers the four history rules: no match without any one of the fields a search needs, a match with them, and every placeholder resolved.
  • A committed, reproducible EventProcessor runner includes eleven fabricated raw cases, common regex definitions, exact field/error assertions and positive/negative alert assertions. It stages only local parser/CEL/file-writer plugins.
  • Full plugins/alerts Go suite on go-sdk v1.1.36: 48 pass, 11 skip (other technologies' private-evidence tests, same as the base), 0 fail, also after the ransomware guard. All 16 source rules compile; boolean compatibility controls cover strings, native booleans and negative types.

Evidence limits: source discovery found no retained Deceptive Bytes documents in 29 successful queries; two discovery attempts failed. These corrections are grounded in SDK/wiki and public parser behavior, not claimed as customer-log validation. The tested EventProcessor commit is 8a3ade72bd9d12db21f6b273200588fb49540f14, with go-sdk v1.1.36 in the playground and every plugin, the version this module pins; it is not asserted to match a customer deployment. Actual vendor event populations, deployed extraction, history, grouping and notifications remain unverified. The source audit documents these limits and reproduction commands.

Replaces closed #2592 with raw-input extraction and executable rule-alert checks addressing its review feedback. Draft against v11 for team review; no deployment or merge is requested.

Hosted CI: the architecture, bugs and security review jobs in run 35892985326 all stop before review at ai-review.sh:29 because THREATWINDS_API_KEY is required. All three job logs were checked. Prompt discovery passed; Go dependency checking was still running. This infrastructure failure is separate from the passing local tests and is not a completed AI review.

🤖 Generated with Claude Code

kryonsx and others added 2 commits September 23, 2026 13:02
Data theft, advanced threat tactic and zero-day history searches filter on
{{.origin.ip}} (and tacticName or processName), but their conditions did not
require those fields and this filter never writes origin.ip. A missing
placeholder makes the search return an error, so the first five matching
records would switch each rule off with a Circuit Breaker alert. Require the
placeholder fields in the conditions, as the other seven origin.ip rules of
this source already do, and test that matching events resolve every
placeholder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kryonsx

kryonsx commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Change pushed (328df641): three history rules no longer switch themselves off when the source address is missing

What changed

  • data_theft_attempt_indicators.yml, advanced_threat_tactic_identification.yml and zero_day_behavior_patterns.yml (all v1.0.1): each condition now also requires the fields its history search uses. That is origin.ip for all three, plus log.processName for zero-day (log.tacticName is already required by the tactic list).
  • New plugins/alerts/deceptive_bytes_history_guard_test.go: checks that each rule matches only when every history placeholder can be resolved.

Why

  • These three history searches filter on {{.origin.ip}}, but this filter never writes origin.ip. When a placeholder has no value, the engine returns "expression value cannot be nil after placeholder resolution". After five such errors it switches the rule off with a "Circuit Breaker" alert.
  • The other seven Deceptive Bytes rules that use origin.ip already require it in their conditions, so this makes the set consistent.

Validation

  • No production instance has Deceptive Bytes data. The check used the 6 fixtures in this draft plus 6 made-up data-theft lines, run through the real plugins (local EventProcessor playground, commit 497bf53, rule engine library v1.1.34).
  • Before the change, all six data-theft lines failed with that error and "Data Theft Attempt on Decoy Files" tripped its Circuit Breaker. After it, there are no errors and no Circuit Breaker.
  • go test ./... in plugins/alerts passes.

Still open: 10 of the 16 rules need origin.ip, which this filter does not produce, so they cannot fire yet. Mapping a source address needs a real Deceptive Bytes log sample or vendor documentation that shows where the source address is. Neither is available to this review.

@kryonsx kryonsx mentioned this pull request Sep 24, 2026
5 tasks
kryonsx and others added 5 commits September 24, 2026 16:36
Brings in go-sdk v1.1.36 for plugins/alerts (field names keep underscores; regexMatch matches strings only) and the coordinated vendor re-baseline. No file overlaps this draft.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-sdk v1.1.36

go-sdk v1.1.35 changed utils.SanitizeField to keep underscores, and v11 now pins v1.1.36, so the KV parser stores Deceptive Bytes keys with their underscores (log.event_type, log.source_ip). The draft had respelled 18 such names in six rules without underscores, as the older SDK stored them; on the latest engine those names never match. Restore the original names in the predicates, history fields and placeholders, and grouping paths. Lateral movement, living off the land, privilege escalation and ransomware are again identical to v11; data theft keeps its origin.ip guard and nation state its "true" comparisons.

The committed fixtures now carry all 18 keys, replay.py checks that each is stored with its underscore, and it stages the three rules without history searches. The Go tests build their keys the way the linked SDK stores them. On EventProcessor 8a3ade7: 11 events, 0 parser errors, 3 intended alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and engine

Official v11 d2479c1 (go-sdk v1.1.36 in plugins/alerts) and EventProcessor main 8a3ade7, the revision in the newest published engine image. Record the restored underscored names, the history guards, the unchanged parser results and the latest test results: full plugins/alerts suite 48 pass, 11 skip, 0 fail; replay.py 11 events and 3 alerts; go-sdk v1.1.36 rule replay 89 of 89 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… placeholders exist

The ransomware rule searches history on {{.log.process}} and {{.log.source_ip}}, but its condition did not require those fields. A missing placeholder makes the search fail, and five failures switch the rule off with a Circuit Breaker alert. Require both fields, as the data theft, advanced threat tactic and zero-day rules already require theirs.

The history-guard test now covers the four rules and checks that a rule does not match without any one of the fields it needs; it fails on the unguarded ransomware rule. go-sdk v1.1.36 replay: the unguarded rule matched two lines that each lacked one field, with unresolved placeholders; the guarded rule matches only the line with both (65 of 65 checks). Full plugins/alerts suite: 48 pass, 11 skip, 0 fail; replay.py on EventProcessor 8a3ade7 still passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… v11 by its guard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@osmontero
osmontero marked this pull request as ready for review September 25, 2026 15:17
@osmontero
osmontero requested a review from a team September 25, 2026 15:17
@osmontero
osmontero merged commit 586098e into utmstack:v11 Sep 25, 2026
3 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants